de dupe url rewrit rule anirban dasgupta ravi kumar amit sasturkar larg fraction url web duplic duplic content de dupe url extrem import problem search engin princip function search engin includ crawl index rank present advers impact presenc duplic url tradition de dupe problem address fetch examin content url approach set url partit equival class base content url equival class similar content address problem mine set learn url rewrit rule transform url equival class canon form rewrit rule appli elimin duplic url encount time crawl fetch content order express transform rule propos simpl framework gener captur common url rewrit pattern occur web encapsul dust iffer rl similar ext framework provid effici algorithm mine learn url rewrit rule show mild assumpt complet algorithm learn url rewrit rule correct notion correct demonstr express framework effect algorithm perform varieti extens larg scale experi 